Nonlinear Feature Transforms Using Maximum Mutual Information

Finding the right features is an essential part of a pattern recognition system. This can be accomplished either by selection or by a transform from a larger number of “raw” features. In this work we learn non-linear dimension reducing discriminative transforms that are implemented as neural networks, either as radial basis function networks or as multilayer perceptrons. As the criterion, we use the joint mutual information (MI) between the class labels of training data and transformed features. Our measure of MI makes use of Renyi entropy as formulated by Principe et al. Resulting low-dimensional features enable a classifier to operate with less computational resources and memory without compromising the accuracy.